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ABSTRACT 

The purpose of this study was to determine a student’s computer knowledge upon course entry and if there was a dif¬ 
ference in college students’ improvement scores as measured by the difference in pretest and post-test scores of new or 
novice users, moderate users, and expert users at the end of a college level introductory computing class. This study also 
determined whether there were differences in improvement scores by gender or age group. The results ofthis study were 
used to determine whether there was a difference in improvement scores among the three campus locations participat¬ 
ing in this study. 

Four hundred sixty-nine students participated in this study at a community college located in Northeast Tennessee. A 
survey, pretest, and post-test were administered to students in a college level introductory computing class. The survey 
consisted ofdemographic data that included gender, age category, location, Internet access, educational experience and 
the self-rated user category, while the pretest and post-test explored the student’s knowledge of computer terminology, 
hardware, the current operating system, Microsoft Word, Microsoft Excel, and Microsoft PowerPoint. 

The data analysis revealed significant differences in pretest scores between educational experience categories. In each 
instance, the pretest mean forfirst semester freshmen students was lower than second semester freshmen and sopho¬ 
mores. The study also reported significant differences between the self-rated user categories and pretest scores as well 
as differences in improvement scores (post-test scores minus pretest scores). However, the improvement scores (post-test 
scores minus pretest scores) were higher than the other self-rated user categories. Of the three participating campus 
locations, students at Location 1 earned higher improvement scores than did students at Location 2. The results also 
indicated that there was a significant difference between the types of course delivery and course improvement scores 
(post-test scores minus pretest scores). The improvement scores for on ground delivery was 5 points higher than the 
hybrid course delivery. Finally, study revealed no significant differences according to the gender and age categories. 


INTRODUCTION 

College level computing skills are useful tools that serve 
students throughout their college career. However, many 
students enter college lacking necessary computing skills. 
While many students might be proficient in locating in¬ 


formation online through search engines, less is known 
about the use and application of specific types of software 
often found in business and industry. As a result of this 
lack of knowledge, all students entering the participating 
community college must prove computer competency ei- 
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ther by taking a competency exam or by the completion 
of a college level introductory computing class. Approxi¬ 
mately 97% of the students chose to take the introductory 
class to satisfy this competency requirement. Assessment 
methods used to evaluate students in the introductory 
class included hands-on project tutorials, a research paper, 
and multiple-choice quizzes. 

The purpose of this study was to determine a student’s 
computer knowledge upon course entry and if there was 
a difference in college students’ improvement scores as 
measured by the difference in pretest and post-test scores 
of new or novice users, moderate users, and expert users 
at the end of a college level introductory computing class. 
This study also determined whether there were differences 
in improvement scores by gender or age group. The results 
of this study were used to determine whether there was a 
difference in improvement scores among the three cam¬ 
pus locations participating in this study. 

This study focused on student improvement in a college 
level introductory computing class using a pretest and a 
post-test at the participating community college. The as¬ 
sessment of the pretests and post-tests and the results of 
these tests were the criterion variables for the study. These 
independent variables included: gender, age, campus loca¬ 
tion, prior higher education experience, residential Inter¬ 
net access, and user’s self-rated computer skill level. 

RELATED LITERATURE 

Within the literature, definitions of computer literacy 
have varied from author to author. Often, individuals’ ac¬ 
tual task-specific computer skills and their perceived com¬ 
puter skills do not coincide (Dettori, Steinbach, & Kalin, 
2006). According to Messineo and DeOllos (2005), high¬ 
er level of experience with forms of technology produced 
more confidence. However, it was suggested that with ad¬ 
vanced applications, the confidence level and the exposure 
level was lacking. Incorrect assumptions are sometimes 
made by faculty members regarding studentpreparedness 
to take the introductory computer science class. 

The term computer literacy has also varied throughout the 
years. For example, what we understand as computer lit¬ 
eracy has assumed different names and meanings since the 
1980s. Definitions were influenced by various theories. 
The National Science Foundation (NSF) hosted a con¬ 
ference in 1980 to discuss the meaning of the term “com¬ 
puter literacy” (Childers, 2003). Burniske (2000) stated 
that, “To prepare ourselves and our students for new types 
of literacy, we must be receptive to new definitions of the 
term itself” (p. 3). Burniske addressed two types of liter¬ 
acy. The first type of literacy was functional literacy. This 
concept was popularized by the United States Army dur¬ 


ing World War II. Functional literacy included the lowest 
functioning level of literacy and rarely required an indi¬ 
vidual to use problem-solving techniques. Functional lit¬ 
eracy focused on teaching the basics of reading and writ¬ 
ing. The second literacy type was critical literacy, which 
often referred to a learned individual with the ability to 
solve problems. This type of literacy comprised teaching 
the student learning to interpret and apply new informa¬ 
tion presented. Many researchers considered computer 
literacy a type of critical literacy. To integrate computer 
literacy, the instructor often blended traditional teaching 
with new technologies. In the classroom, teachers are of¬ 
ten required to move beyond simply teaching a skill, such 
as keyboarding, to integrating computer skills within the 
core curriculum. This required the teacher to have a com¬ 
bination of a technical skill set and a theory based skill set. 
According to Burniske, if we are to achieve literacy-across- 
the curriculum, formal teacher training is required. 

Computer skills considered necessary for computer litera¬ 
cy varied according to position. For instance, students as¬ 
sumed computer literacy if they could play games or word 
process a document, activities important to them, thus 
producing self-efficacy. Self-efficacy included one’s belief 
in their skill for successful task completion. Individuals 
who reported high levels of self-efficacy tended to face 
difficult challenges more easily than others. Additionally, 
individual beliefs affected how persons felt, behaved, and 
motivated themselves (Bandura, 1997). 

Technology skills assessments have taken many forms. For 
instance, Martin and Dunsworth (2007) proposed for¬ 
mative assessment of computer literacy at the university 
level to improve curriculum design of a computer literacy 
course. This formative assessment included the technolog¬ 
ical advances of the workplace as well as the technologi¬ 
cal needs of the student. Class observations, student test 
scores, student and teacher focus groups, and instructor 
surveys were tools used to collect the data. Four hundred 
forty-four students received a Likert-type survey through 
the Blackboard Course Management System in which 
329 students responded. The researchers interviewed five 
focus groups comprised of 25 students as well as the 11 in¬ 
structors who delivered the course. Five class observations 
also aided in data collection. The compiled data formed 
two categories: 1) what to teach, and 2) how to teach it. 
The findings reported both instructors and students rat¬ 
ed Microsoft Office Skills, particularly Word and Pow¬ 
erPoint, as necessary. Additionally, both groups agreed 
that in class activities and hands-on projects were useful 
approaches when teaching computer literacy. Instructors 
and students stated that the Internet and the World Wide 
Web were considered important tools. However, students 
reported that online quizzes and extended lectures were 
not helpful, while instructors deemed them valuable 


teaching tools and a means to measure student learning. 
Neither students nor instructors considered knowledge of 
computer hardware (input, processing, storage, and out¬ 
put) as a necessary skill. Instructors submitted that File 
Management was a needed skill, while students assigned 
a lower rating to this skill. Recommendations from the 
study included the need for more in class and hands-on 
activities, and collaborative activities that provided a 
group learning atmosphere. 

Several higher education institutions adopted computer 
literacy requirements. For example, in 2010 Cape Fear 
Community College (CFCC) in North Carolina man¬ 
dated that students prove computer competency to gradu¬ 
ate. The students were presented with two options which 
satisfied competency requirements. They must have suc¬ 
cessfully passed the computer competency exam, a one 
hour exam, or have completed a designated college trans¬ 
fer computer course. If students chose the proctored com¬ 
petency exam, it was administered through Blackboard, a 
course management software application. In preparation 
for the exam, CFCC provided a computer competency 
tutorial and a computer competency practice exam for 
students (Cape Fear Community College, 2010). 

RESEARCH METHODOLOGY 

Research Questions 

The following research questions guided this study: 

1. Are there significant differences in students’ 
pretest scores among the three college experience 
categories (freshman - 1st semester, freshman 

- 2nd semester, and sophomore- 1st and 2nd 
semester) in college level introductory comput¬ 
ing classes? 

2. Are there significant differences in students’ pre¬ 
test scores among the five types of self-reported 
residential Internet access (dial-up, cable, DSL, 
wireless and no Internet access) in college level 
introductory computing classes ? 

3. Are there significant differences in students’ 
pretest scores among the three self-rated user 
categories (new or novice user, moderate user, 
and expert user) in college level introductory 
computing classes ? 

4. Are there significant differences in students’ im¬ 
provement scores (post-test scores minus pretest 
scores) among the three self-rated categories 


(new or novice user, moderate user, expert user) 
in college level introductory computing classes ? 

5. Are there significant differences in students’ 
improvement scores (post-test scores minus 
pretest scores) among the three campus locations 
(Campus Location 1, 2, and 3) in college level 
introductory computing classes ? 

6. Are there significant differences in students’ im¬ 
provement scores (post-test scores minus pretest 
scores) among the three age categories (age 15- 
19, age 20-28, age 29 and older) as determined by 
gender in college level introductory computing 
classes ? 

7. Are there significant differences in students’ im¬ 
provement scores among the three self-rated user 
categories (new or novice user, moderate user, 
expert user) and the three age categories (age 15- 
19, age 20-28, age 29 and older) in college level 
introductory computing classes ? 

8. Are there significant differences in students’ 
improvement scores (post-test scores minus 
pretest scores) among the course delivery types 
(on ground courses, online courses, and hybrid 
courses) in college level introductory computing 
classes ? 

Population 

Students from 26 sections of the introductory computer 
science course participated in the study. In each section, 
the instructor administered the pretest, post-test, and 
survey to those students who had chosen to participate. 
A total of400 students, out of a potential 426, completed 
both the pretest and the post-test. The participating com¬ 
munity college served ten surrounding counties with 
three campuses serving diverse populations. Students 
from three geographically unique campuses participated 
in this study. The campus locations in the study included: 
Location 1, centrally located; Location 2, located furthest 
southeast of the campuses; and Location 3, located fur¬ 
thest south. Because all course sections administered the 
pretest, post-test, and the survey, there was no skewing of 
the data by either the selection of a particular introduc¬ 
tory computer science course or the time designation that 
each course was offered. 
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Instrumentation 

A group of Computer Science instructors at the partici¬ 
pating college aided in the development of the pretest and 
post-test. The questions represented each unit studied 
throughout the course. Administration of the pretest and 
post-test were managed through the course management 
system and consisted of 100 questions. The questions in¬ 
corporated the chapter units of the course, as follows: (a) 
Chapters 1-3, operating system; (b) Chapters 1-4, basic 
word processing; (c) Chapters 1-4, basic spreadsheet chap¬ 
ters; (d) Chapters 1 and 2, basic presentation software. 

The student survey instrument contained various demo¬ 
graphic questions. The independent variables included: 
gender, age, college experience, campus location, residen¬ 
tial Internet access, and the user’s self-rated computing 
skill level. The survey questions were comprised of mul¬ 
tiple choice answers. One particular survey question re¬ 
garding the user’s self-rated computing skill level was of 
particular importance to this study. The question required 
the participants to read descriptions of each of three de¬ 
fined categories. They then selected the category that best 
described their computing skill level. The three selfrated 
categories were new or novice user, moderate user, and ex¬ 
pert user. Because the demographic survey was optional, 
some students chose not to participate in this portion of 
the study or they completed only portions of the survey. 

Data Collection 

The online course management system used in the study 
was Desire to Team. The online course management sys¬ 
tem provided one central location for course materials, 
quizzes, surveys, calendars, and drop boxes for students 
to submit assignments with no installation of additional 
software required by the participants. The data provided 
for the study were collected through the course manage¬ 
ment system by a designee of the division dean. 

In addition, a demographic survey was administered elec¬ 
tronically along with the pretest .The demographic survey 
was developed with the assistance of the instructors in the 
Computer Science Department. Each instructor of the 28 
participating course sections explained the purpose of the 
survey to each class and noted that student participation 
was optional. As with the pretest and post-test delivery, 
the demographic survey was administered electronically 
as part of the class through the course management sys¬ 
tem. Students logged in to the course management system 
and entered into their college level introductory comput¬ 
ing class to take the survey located in the Surveys section 
of the course. If students chose to participate, students 
were then instructed to complete and submit the demo¬ 
graphic survey questions electronically. Data provided by 


the students in the study were used only for the purposes 
of this study and the Computer Science Department of 
the participating community college. Pretest, post-test, 
and survey data were collected by a designee of the divi¬ 
sion dean to protect the anonymity of students who chose 
to participate in the study. 

RESULTS 

Research Question 1 

A one-way analysis of variance was used to evaluate the re¬ 
lationship between students’ pretest scores and the college 
experience of students enrolled in college level introduc¬ 
tory computing classes. The dependent variable was pre¬ 
test scores. The independent variable, college experience, 
had three levels: first semester freshmen, second semester 
freshmen, and sophomores - first and second semester. 
The ANOVAwas significant, F(2, 423) = 11.01 ,p <.001. 
The effect size as measured by q 2 was small (.05). That is, 
5% of the variance in students’ pretest scores was account¬ 
ed for by college experience. 

Because the overall F test was significant, multiple post 
hoc comparisons were conducted to evaluate pairwise 
differences in the pretest means of the three groups. The 
Tukey post hoc test was used because equal variances were 
assumed, F (2,423) = .85 ,p = .430. The Tukey procedure 
determined that there was a significant difference between 
first semester and second semester freshmen (p <.001) and 
between first semester freshmen and sophomores - first 
and second semester (p = .020). In each instance, the pre¬ 
test mean for first semester freshmen students was lower. 
The pretest mean for first semester freshmen was over six 
points lower than the mean for second semester freshmen 
and over 3.5 points lower than the mean for sophomores - 
first and second semester. There was no significant differ¬ 
ence between second semester freshmen and sophomores 
- first and second semester (p - .322). 

Research Question 2 

A one-way analysis of variance was used to evaluate the 
mean differences in students’ pretest scores among the five 
types of self-reported residential internet access. The de¬ 
pendent variable was the pretest scores. The independent 
variable, type of residential internet access, had five levels: 
dial-up; cable; DSL, wireless and no internet access. The 
ANOVA was not significant, F (4, 421) = 1.48, p = .209. 
The effect size as measured by vp was small (.01). That is, 
only 1% of the variance in pretest scores was accounted for 
by the type of internet access. The results indicated that 
the type of residential internet access did not significantly 
affect students’ pretest scores. 


Research Question 3 

A one-way analysis of variance was completed to evaluate 
the relationship between students’ pretest scores and the 
self-rated user category in college level introductory com¬ 
puting classes. The dependent variable for this ANOVA 
model was the pretest scores. The independent variable, 
self-rated user category had three levels: new or novice 
user, moderate user, and expert user. The ANOVA was 
significant, F ( 2, 422) = 40.74 ,p <.001. The effect size as 
measured by V] 2 was large (.16). That is, 16% of the vari¬ 
ance in pretest scores was accounted for by self-rated user 
category. 

Because the overall F test was significant, follow up tests 
to evaluate the differences among the pairs of pretest 
means were conducted. The Tukey post hoc test was used 
because equal variances were assumed, F (2, 422) = .78, 
p = .459. The Tukey procedure determined that all pairs 
of pretest means were significantly different at p <.001. 
In each pair of means evaluated, the lower the self-rated 
user level had the lower pretest mean. That is, the pretest 
mean for self-rated new or novice users was over 7.0 points 
lower than self-rated moderate users and almost 15 points 
lower than self-rated expert users. The pretest mean for 
self-rated moderate users was 7.4 points lower than self- 
rated expert users. 

Research Question 4 

A one-way analysis of variance was completed to evaluate 
the relationship between students’ improvement scores 
and the self-rated user category in college level introduc¬ 
tory computing classes. The dependent variable was im¬ 
provement scores. The independent variable, self-rated 
user category had three levels: new or novice user, moder¬ 
ate user, and expert user. The ANOVA was significant, F 
(2,372) = 15.54,p <.001. The effect size as measured by q 2 
was medium (.08). That is, 8% of the variance in improve¬ 
ment scores was accounted for by self-rated user catego¬ 
ries. 

Because the overall F test was significant, post hoc multi¬ 
ple comparisons were conducted to evaluate which pair of 
improvement score means was different. Levene’s Test of 
Equality of Error Variances showed equal variances could 
not be assumed, F (2, 372) = 4.33 ,p = .014. Therefore, the 
Dunnett’s C post hoc test was used to test pairwise differ¬ 
ences. All three pairs of means were significant at the .05 
level. Self-rated new or novice users’ improvement score 
mean was 5.6 points higher than self-rated moderate users 
and 10 points higher than self-rated expert users. Moder¬ 
ate users’ mean improvement was 4.5 points higher than 
expert users. 


Research Question 5 

A one-way analysis of variance was completed to evaluate 
the differences in students’ improvement scores among 
the three campus location in college level introductory 
computing classes. The dependent variable was improve¬ 
ment scores. The independent variable, campus locations 
had three levels labeled: Location 1, Location 2, and Lo¬ 
cation 3. The ANOVA was significant, F (2, 369) = 3.57, 
p =.029. The effect size as measured by q 2 was small (.02) 
indicating that 2% of the variance in improvement scores 
was accounted for by campus location. 

Because the overall F was significant, multiple post hoc 
comparisons were conducted to determine which pair of 
means was different. Dunnett’s C was used because equal 
variances were not assumed, F (2, 369) = 6.03 ,p = .003. 
Dunnett’s C showed there was a significant difference in 
improvement score means between Location 1 and Lo¬ 
cation 2. The improvement mean for Location 1 was 3.1 
points higher than the mean for Location 2. No other 
pairs of means were significantly different. 

Research Question 6 

A two-way ANOVA was used to determine if any signifi¬ 
cant differences in improvement scores between any of the 
three age categories. The ANOVA showed there was no 
significant age by gender interaction, F (2,370) = .536,p = 
.585. The effect size as measured by q 2 was small (<.01) in¬ 
dicating that less than 1% of the variance in improvement 
scores was accounted for by age by gender interaction. 
There was no significant difference in the improvement 
score means among the age categories, F (2, 370) = 2.966, 
p = .057. The effect size as measured by q 2 was small (.02) 
indicating that 2% of the variance in improvement scores 
was accounted for by age. Finally, there was no significant 
difference in improvement score means between male and 
female students, F(\, 370) = .489, p = .485. The effect size 
as measured by q 2 was small (<.01). That is, less than 1% 
of the variance in improvement scores was accounted for 
by gender. 

Research Question 7 

A two-way ANOVA was used to determine if there were 
differences in students’ improvement score means based 
on age and self-rated user categories in college level in¬ 
troductory computing classes. The ANOVA showed that 
there was no significant two-way interaction between age 
by self-rated user category, F (4, 366) = .61 ,p = .653. The 
effect size for the interaction term as measured by q 2 was 
small (.01).The ANOVA also revealed that age categories 
were not significant, F (2, 366) = 1.80, p = .167. The ef¬ 
fect size as measured by q 2 was small (.01). That is, 2% of 


Spring 2014 (Volume 10 Issue 1) 


Journal of Learning in Higher Education 


62 


63 



the variance in improvement scores was accounted for by 
age. However, the self-rated user category was significant, 
F(2, 366) = 12.54,^? <.001. The effect size as measured by 
if was medium (.06) indicating that 6% of the variance 
in improvement scores was accounted for by the self-rated 
user category. 

Regarding the significance of the self-rated user category, 
as reported in the discussion of Research Question 4, 
Dunnett’s C showed all three pairs of improvement score 
means were significant at the .05 level. New or novice us¬ 
ers’ improvement score mean was over 5.5 points higher 
than moderate users and 10 points higher than expert 
users. Moderate users’ mean improvement was 4.5 points 
higher than expert users. 

Research Question 8 

A one-way analysis of variance was completed to evaluate 
the relationship between students’ improvement scores 
among the course delivery types in college level introduc¬ 
tory computing classes. The dependent variable was im¬ 
provement scores. The independent variable, course deliv¬ 
ery type had three levels: on ground, online and hybrid. 
The ANOVAwas significant, F (2, 397) = 3.36 ,p = .036. 
However, the effect size as measured by r] 2 was small (.02) 
indicating that 2% of the variance in improvement scores 
was accounted for by the course type. 

Because the overall F was significant, multiple pair¬ 
wise comparisons were conducted to determine 
which pair of means was significant. The Tukey test 
was used because equal variances were assumed, F (2, 
397) = 1.49,^ = .226. The Tukey procedure determined 
that there was a significant difference in the improve¬ 
ment means between on ground and hybrid courses 
(p = .048). The improvement score mean for on ground 
courses was five points higher than the mean for hybrid 
courses. However, there was no significant difference be¬ 
tween on ground and online course (p = .447) and no 
significant difference between online and hybrid courses 
0 P = -801). 

SUMMARY AND RECOMMENDATIONS 

Onsite, online and hybrid courses comprised the methods 
of course delivery available to students. The findings re¬ 
vealed no significant difference between mean students’ 
improvement scores (post-test scores minus pretest scores) 
in the on ground and online courses. However, there was a 
significant difference between improvement scores in the 
on ground and hybrid courses. Mean improvement scores 
for on ground courses were 21% higher than hybrid cours¬ 
es and 13% higher than online courses. One potential rea¬ 
son for this disparity could be that instructors clarify class 


concepts and assignments for on ground courses with 
just-in-time teaching, while online courses might require 
several communications to explain an instruction or as¬ 
signment. 

Advanced, detailed knowledge of course delivery meth¬ 
ods would provide additional information for the student 
before they registered for a course. The institution would 
benefit from the creation of an online columnar table of 
delivery types. The table would detail specific components 
included in each course type, on ground, online, and hy¬ 
brid. This would provide better understanding when reg¬ 
istering for courses, thus improving a student’s success 
rate in the course. The participating community college 
should continue to standardize course requirements for 
all sections of the college level introductory computing 
class to ensure quality for students. Each college level in¬ 
troductory computing class should continue to adminis¬ 
ter an exit survey to elicit student feedback. 


For new or novice users, the college level introductory 
computing class should provide a “first steps” video library 
embedded in D2T. Camtasia (http://www.camtasia.com) 
or Jing (http://www.jing.com) are two common editing 
software packages used to create videos. Some introduc¬ 
tory video topics would include opening and closing a file, 
saving a file to different storage locations, and download¬ 
ing and extracting a file from the course management sys¬ 
tem. Students could also be directed to free resources that 


are available online to increase a student’s initial comput¬ 
ing skill level. In 2009, Microsoft established the Micro¬ 
soft Digital Titeracy Program. This program is comprised 
of a series of videos that teach standard literacy skills. The 
Standard Skills Curriculum includes computer basics, 
the Internet an introduction to productivity software, se¬ 
curity, and leading a digital lifestyle to build computing 
self-efficacy. As Orr, Allen and Poindexter (2001) stated, 
instructors could apply interventions if they had better 


understanding of the computer attitudes of their students. 


Adobe Connect (http://www.adobe.com) is another way 
to link with students through the use of technology. The 
purchase and use of Adobe Connect web conferencing 
software in a college level introductory computing class 
would facilitate more immediate feedback for online and 


hybrid students while providing student engagement data 
for the instructor. 


The participating college should develop course learning 
modules for the college level introductory computing class 
to tailor student learning. These course learning modules 
are units of study that students could complete within 
a specified time period at their own pace and with little 
instructor interaction. In order for students to move for¬ 
ward to the next module, they would have to attain a pre¬ 
determined minimum module score. For self-rated expert 
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users, this would provide an alternative to the traditional 
classroom instruction. 

Connected Tennessee’s (http://www.connectedtn.org) 
organizational mission statement emphasizes design 
strategies to educate, use, and deliver technology access to 
Tennesseans. Tocation 2 would continue to benefit from 
expanded broadband connectivity for its rural users. 
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